The effects of expertise and an eye movement cue on self-generated self-assessment criteria

This study investigates whether replays of records of eye movements made during the task performance process, can be a useful cue for self-assessment, and it aims to uncover which aspects of their performance participants at different levels of expertise consider when they are asked to engage in self-assessment while thinking aloud.The present study suggests that at least for somewhat advanced individuals, performance process cues could stimulate self-assessment, but that this does not apply to novices. This study investigated whether replays of records of eye movements made during the task performance process could be a useful cue for self-assessment. Self-assessment is very difficult for learners, especially novice learners, due to their lack of expertise. On the one hand, they experience a high cognitive load imposed by the learning tasks (Sweller, Van Merriënboer, & Paas, 1998; Van Merriënboer & Sweller, 2005), which may make it difficult to monitor their performance. On the other hand, even if they would be able to monitor their performance, they might fall prey to the “Double Curse of Incompetence”, that is, “... skills necessary to recognize competence are extremely close if not identical to those needed to produce competent responses” (Dunning, Heath, & Suls, 2003, p. 73). This study, in the domain of biology, aimed to uncover which aspects of their performance participants at different levels of expertise considered when they were asked to engage in self-assessment while thinking aloud, either with or without a cue of the performance process. The cue used here, was a replay of a record of eye movements and mouse and keyboard operations made during task performance (cf. the cue used by Van Gog, Paas, Van Merriënboer, & Witte, 2005; see Figure 1.). Participants were given no lists of criteria, because we intended to investigate what they spontaneously considered when engaging in self-assessment. It was hypothesized that: a) the cue is helpful for novice participants to report about the performance process, whereas advanced participants (higher expertise) do not need a cue to be able to report on the process, and b) the cue may or may not help novices evaluate their performance process, but will indeed help advanced participants (with higher expertise) evaluate their performance, which will show not only in enhanced frequency of evaluations, but also in the criteria used. Forty adults volunteered to participate in this study. A 2 x 2 factorial design was used with ‘Expertise’ (Lower vs. Higher) and ‘Self-assessment’ (Cued vs. Non-cued). After each of the four tasks, participants were required to give a self-assessment of their performance, while thinking aloud (Ericsson & Simon, 1993), either with or without the cue depending on their assigned condition. The cue was recorded and replayed with a Tobii 1750 Eye Tracker running at 50Hz, using the screen capture mode in the Clearview 2.7.1 software. Participants’ verbalizations were recorded using Audacity 1.2.6 software. After the experiment, the verbal protocols were transcribed and segmented based on meaningful units (i.e., partial, whole, or multiple sentences). The segments were coded using a coding scheme with three main categories: a) Survey, b) Action, and c) Monitoring/Assessment. The category Monitoring/Assessment was further divided into four “criteria”: 1) Adequacy: further divided into being either positive or negative; 2) Efficiency; 3) Affect, also divided into being either positive or negative, and 4) Difficulty. In accordance with the hypotheses, the results seem to suggest that lower expertise participants indeed have difficulty remembering the details of the performance process due to higher cognitive load, and that the cue allows them to review and remark on those task processes. However, these findings also provide support for the assumption that some expertise is required in order to be able to evaluate ones own performance and that the cue can be a helpful support tool under those conditions. In general, though, not many of the possible selfassessment criteria were used, especially by the lower expertise participants. The present study suggests that at least for somewhat advanced individuals, performance process cues could stimulate self-assessment, but that this does not apply to novices. Figure 1. Example of a Cue. Participants saw a red dot indicating their fixations moving across the screen, followed by a 1000 ms gaze trail indicating the movement during the last second.